Skip to content

chore: Enable automation testing#3260

Open
stevehipwell wants to merge 1 commit intomainfrom
automation-org
Open

chore: Enable automation testing#3260
stevehipwell wants to merge 1 commit intomainfrom
automation-org

Conversation

@stevehipwell
Copy link
Copy Markdown
Collaborator

Resolves #ISSUE_NUMBER


Before the change?

  • App auth required setting the app_auth block
  • Automation test couldn't be run due to a lack of infrastructure

After the change?

  • App auth can be configured with only environment variables
  • Automation tests run for non-fork PRs
  • Automation tests run for fork PRs with the acctest label (they currently need to be approved)
  • Automation tests run for pushes to main

Pull request checklist

  • Schema migrations have been created if needed (example)
  • Tests for the changes have been added (for bug fixes / features)
  • Docs have been reviewed and added / updated if needed (for bug fixes / features)

Does this introduce a breaking change?

Please see our docs on breaking changes to help!

  • Yes
  • No

@stevehipwell stevehipwell added this to the v6.12.0 Release milestone Mar 6, 2026
@stevehipwell stevehipwell requested a review from deiga March 6, 2026 15:32
@stevehipwell stevehipwell self-assigned this Mar 6, 2026
@stevehipwell stevehipwell added the Type: Maintenance Any dependency, housekeeping, and clean up Issue or PR label Mar 6, 2026
@github-actions
Copy link
Copy Markdown

github-actions bot commented Mar 6, 2026

👋 Hi! Thank you for this contribution! Just to let you know, our GitHub SDK team does a round of issue and PR reviews twice a week, every Monday and Friday! We have a process in place for prioritizing and responding to your input. Because you are a part of this community please feel free to comment, add to, or pick up any issues/PRs that are labeled with Status: Up for grabs. You & others like you are the reason all of this works! So thank you & happy coding! 🚀

@stevehipwell stevehipwell had a problem deploying to acctest-dotcom-untrusted March 6, 2026 15:32 — with GitHub Actions Error
@stevehipwell stevehipwell had a problem deploying to acctest-dotcom-untrusted March 6, 2026 15:32 — with GitHub Actions Failure
@stevehipwell stevehipwell had a problem deploying to acctest-dotcom-untrusted March 6, 2026 15:37 — with GitHub Actions Error
@stevehipwell stevehipwell had a problem deploying to acctest-dotcom-untrusted March 6, 2026 15:37 — with GitHub Actions Error
@stevehipwell stevehipwell had a problem deploying to acctest-dotcom-untrusted March 6, 2026 15:42 — with GitHub Actions Error
@stevehipwell stevehipwell had a problem deploying to acctest-dotcom-untrusted March 6, 2026 15:42 — with GitHub Actions Failure
@stevehipwell stevehipwell had a problem deploying to acctest-dotcom-untrusted March 6, 2026 15:45 — with GitHub Actions Error
@stevehipwell stevehipwell had a problem deploying to acctest-dotcom-untrusted March 6, 2026 15:45 — with GitHub Actions Error
@stevehipwell stevehipwell had a problem deploying to acctest-dotcom-untrusted March 6, 2026 15:50 — with GitHub Actions Failure
@stevehipwell stevehipwell had a problem deploying to acctest-dotcom-untrusted March 6, 2026 15:59 — with GitHub Actions Failure
@stevehipwell stevehipwell had a problem deploying to acctest-dotcom-untrusted March 6, 2026 16:10 — with GitHub Actions Error
@stevehipwell stevehipwell had a problem deploying to acctest-dotcom-untrusted March 6, 2026 16:11 — with GitHub Actions Error
Comment on lines +9 to 10
# pull_request_target:
pull_request:
Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# pull_request_target:
pull_request:
pull_request_target:

This needs changing before we merge, but it needs to be pull_request to run the tests in the PR before the code is merged.

deiga
deiga previously approved these changes Mar 6, 2026
Copy link
Copy Markdown
Collaborator

@deiga deiga left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you feel so inclined, you could refactor the Check blocks you modified into ConfigStateChecks.

Otherwise LGTM

Copy link
Copy Markdown
Contributor

@austenstone austenstone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left one inline note on the pull_request_target scaffold recommending its removal, since it nudges future edits toward a privileged trigger that would be easy to misuse in a secrets-bearing acceptance-test workflow.

@tag-assistant
Copy link
Copy Markdown

Taking a step back — the goal here is great. External contributors need to be able to run acceptance tests without maintainers doing it manually. But I think we're building a lot of custom machinery (fork detection, label gates, secret validation, untrusted environments) to solve a problem GitHub already solves natively.

Environment protection rules with required reviewers replace almost all of this:

  • Workflow triggers on fork PRs normally
  • Job references an environment with "required reviewers" enabled
  • Workflow pauses and waits for a maintainer to click "Approve"
  • Only then do secrets get injected and tests run
  • New commits on the PR = new deployment = automatically requires re-approval

That last point is critical. The label-based approach has a gap where an attacker can push new commits after the acctest label is applied, and tests re-run without re-review. Environment protection rules close that gap by design.

It also eliminates the interaction risk with things like AI-powered labelers (#3272) — if an automated tool accidentally applies acctest to a fork PR, tests run with secrets and no human ever reviewed the code. With environment protection rules, labels aren't a security boundary at all.

The simplified workflow would look something like:

on:
  pull_request:
  push:
    branches: [main, release-v*]

jobs:
  test:
    runs-on: ubuntu-latest
    environment: acctest-dotcom  # required reviewer gate handles everything
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-go@v5
      - run: go test ./github -v -count=1 -parallel=20 -timeout=30m
        env:
          TF_ACC: "1"
          GITHUB_TOKEN: ${{ secrets.GH_TEST_TOKEN }}
          # ... other test vars

The entire setup job, fork detection, label check, and secret validation step can go away. GitHub's native environment protection is purpose-built for exactly this use case — gating secret access behind human approval on a per-run basis.

@deiga
Copy link
Copy Markdown
Collaborator

deiga commented Mar 22, 2026

I quickly looked into some of the test failures:

  • The App is lacking permissions for Codespaces secrets(?) or just Codespaces in general
  • The App is lacking permission to read Org IP Allowlist
  • "Deploy keys are disabled for this repository" sounds like the Org has disabled Deploy keys?
  • The test run ran longer than 60min => App Token invalid?
  • TestAccGithubActionsOrganizationWorkflowPermissions is failing due to trying to modify values disabled by Enterprise
  • TestAccGithubActionsRunnerGroup should maybe ignore etag in import test

Signed-off-by: Steve Hipwell <steve.hipwell@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Type: Maintenance Any dependency, housekeeping, and clean up Issue or PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants